Error concealment apparatus and method

ABSTRACT

This invention provides a method and apparatus for dynamically selecting and applying different error concealment techniques at run-time on the receiver side. A plurality of error concealment algorithms is stored and dynamically selected for error concealment. A selection signal determines which algorithm is selected. The selection signal is based on parameters denoting computer processing capacity and audio signal characteristics.

FIELD OF TECHNOLOGY

The present invention relates to an error concealment method and to anapparatus for selecting an error concealment method for concealingerrors in real-time streaming signals received in a packet network wheredropped packets are a cause of problems on the receiver side.

DESCRIPTION OF RELATED ART

Much research has been directed to error concealment technologies foraudio data. These methods can generally be classified as methods forreceiver-side reconstruction of dropped packets, sender-sidereconstruction of dropped packets, and methods for reconstructingdropped packets on the sender and receiver sides. See, for example,references [1]to [4] below. [1] O. J. Wasen, D. J. Goodman, C. A. Dvorakand H. G. Page, The Effect of Waveform Substitution on the Quality ofPCM Packet Communications. IEEE Transactions on Acoustics, Speech andSignal Processing, Vol.36, No 3, March 1986. [2] Colin Perkins, OrionHodson and Vicky Hardman, A Survey of Packet Loss Recovery Techniquesfor Streaming Audio, IEEE Network, September/October 1998. [3] R. AValenzuela and C. N. A Nimalu, A new Voice Packet ReconstructionTechnique. [4] V. Hardman, A. Sasse, M. Handley and A. Watson, ReliableAudio for Use over Internet, Proc INET'95. [5]3GPP TS 26.094 V4.0.0(2001-03). [6]Petr Pollak, Pavel Sovka and Jam Uhlir, CepstralSpeech/Pause Detectors. [7], Werner Verhelst and, March Roelands, AnOverlap Technique based on Waveform Similarity (WSOLA) for High QualityTime-Scale Modification of Speech, 1993 IEEE.

Reconstructing dropped packets on the receiver side is easier thanreconstructing dropped packets on the sender side, but is only effectivewhen the frequency of packet loss is low and the packet size is small.

Furthermore, reconstructing dropped packets on the sender side requiresadding redundancy to the transmission packets, that is, addingadditional information to the data packets, thus requiring additionalbandwidth or a longer end-to-end delay. See references [1] and [2]above.

Adding a large amount of extra information is required in methods forreconstructing dropped packets on the sender side. As described inreferences [4]and [7] above, adding additional information includesadding a sequence number to each packet, adding a resend request when apacket is dropped, or adding a low bitrate encoding request for resentpackets, and in any case is complicated.

Methods for reconstructing dropped packets on the sender and receiversides seek to gain the benefits of both receiver-side reconstruction andsender-side reconstruction, but the error concealment technique isstatic and cannot be changed at run-time.

SUMMARY OF THE INVENTION

An object of the present invention is therefore to enable the receiverto dynamically select the best error concealment method at run-time.

Intervals where packets are dropped must be filled with the mostappropriate substitution possible. In a voice packet network this filldata could be silence or data reconstructed based on a best-guessestimate or environmental factors. Because different error concealmenttechniques are more effective with different loss patterns, inserting adynamic stelector before error concealment is applied to evaluate theloss pattern and other environmental factors yields better results.

Furthermore, based on the loss error pattern and such current conditionsas the network and resource level, the present invention enables thereceiver to select the most appropriate method and thereby maximize useof known error concealment methods.

The present invention also provide selection criteria for determiningthe error concealment technique used. The IP protocol currently used onthe Internet is a connectionless, best-effort protocol that does notprovide any quality of service guarantee. Although the best errorconcealment method can be used to handle burst loss in real-timestreaming applications, applying the method of the present inventionwhen congestion occurs or the receiving device suffers a sudden drop inavailable resources makes it possible to switch dynamically duringrun-time to a less resource-intensive computation method.

The present invention also provides an apparatus for concealing errorsusing different methods for packet disassembly.

Yet further, this invention also provides a method that uses thecharacteristics of the loss pattern to decide which error concealmenttechnique to apply. If a voice signal is detected before packet loss ina voice-over-IP (VoIP) application, a signal that is reconstructed frominformation in neighboring signals will be used. If a non-voice signalis detected before packet loss, then either low power noise segments orsilence is used.

To solve these problems, an error concealment apparatus according to thepresent invention uses current resource level information and droppedpacket information. An error concealment apparatus according to thepresent invention has a voice signal detector for detecting segments ofgood signals, whether the signals are voice or non-voice signals. Avoice signal detector of this type is taught in references [5] and [6],for example.

The error concealment method applied to voice signals is selected from aset of usable error concealment algorithms. The selection standards arethe current activity level of the receiver and the level of resourcescurrently available at the receiver. If the available resources are low,an error concealment method that consumes virtually no CPU resources isused. A specific threshold level is assigned to each algorithm based onthe complexity of the algorithm.

Another set of algorithms each having a threshold level set according toeach algorithm is also provided for non-voice signals.

The invention also has a selector module for selecting the besttechnique for concealing errors in a packet network. This selectormodule uses the results passed from other modules such as the voicedetection module. The voice detection module searches template data forvoice segments and non-voice segments. The template data is the datapacket from a preceding packet. The set of error concealment methods isused for processing voice segments and non-voice segments. The selectoralso determines whether to use a simpler algorithm with resources arelow with considering for the existing environmental conditions. Twomethods are used for analysis in order for the error concealment methodto work efficiently, specifically, short-term analysis and long-termanalysis. Short-term analysis is done based on error loss in the audiopackets buffered by the selector module. Long-term analysis is donebased on knowledge of packet loss sent to the receiver, and statisticscollected by the receiver or reliable network nodes.

A first selection apparatus according to the present invention selectsthe error concealment method that is best for concealing lost packets indigital audio signal distributed over a packet switching network so thatthe receiver can decode the digital audio signals in real time. Thisapparatus comprises a packet audio buffer used for packet buffering, aloss pattern detector used to determine the pattern of dropped packetsthat should have been received within a defined time frame, an audiosignal analyzer for extracting the temporal characteristics of adigitized audio signal, and an error concealment selector for selectingthe ultimately best error concealment algorithm based on the resultsreturned by the packet audio buffer, loss pattern detector, and audiosignal analyzer.

Preferably, the selection apparatus measures the packet size denotingthe length (byte count or bit count, for example) of the digital signalencapsulated by the data packet; the number of consecutive loss packetsused to encapsulate in a unit time digital signals of a fixed length orvariable length, and the packet interval representing the digitalsignals measured in the time between two dropped packets; or the actualsignals corresponding thereto (bit count or other).

Furthermore, the loss pattern detector reports generating error packetsin the audio or voice stream using a format composed of the packet size,number of consecutive dropped packets, and the packet interval, oreffectively corresponding signals.

A first method according to the present invention is a method fordetermining characteristics of dropped packets in the packet audiobuffer and loss pattern detector. This method has steps of: bufferingthe previous good packets that encapsulate a fixed or variable length ofdigitized signals in unit time; buffering the next good packets thatencapsulate a fixed or variable length of digitized signals in unittime; determining the number of successive loss packets that encapsulatea fixed or variable length of digitized signals in unit time;determining the pattern of the previous good packets that encapsulate afixed or variable length of digitized signals in unit time; anddetermining the pattern of the subsequent good packets that encapsulatea fixed or variable length of digitized signals in unit time.

A second method of the invention provides a description of the errorloss pattern by providing numerical values for the digital signal losscomputed or measured over an arbitrary period of time by generatingstatistics in unit time or other representation of the specified periodof time where the statistic comprises one of the following: the maximuminterval between digital signals with error or digital signals with loss(referred to below as “digital signals with error or loss”) where theinterval between loss is defined as a measure of distance in terms oftime or packet unit between two instances of digital signal loss, orburst loss is defined as loss of consecutive digital signals in a unittime or other specified time period (including variable); the meaninterval between erroneous or dropped digital signals; the last intervalperiod between digital signals with error or loss measured from thefirst instance of the last batch of consecutive units of non-erroneousdigital signals to the end of the specified period for error patterndescription; the first interval between digital signals with error orloss measured from the beginning of the specified period for errorpattern description to the beginning of the first instance of digitalsignals with error; the maximum burst interval of digital signals witherror or loss, where burst loss is defined as continuous loss of digitalsignals in unit time or other specified period; mean burst of digitalsignals with error or loss; last burst period of digital signals witherror or loss measured from the last batch of consecutive units ofdigital signals with error to the end of the specified period for errorpattern description; and first burst period of digital signals witherror or loss measured from the beginning of the specified period forerror pattern description to the beginning of the first instance of adigital signal without error.

A third error concealment method according to the present invention is amethod for selecting an error concealment algorithm. This methodcomprises steps of; determining the current computational resources forerror concealment; comparing the current resources against the differenterror concealment algorithms based on the error concealment calculationtime and resource consumption; determining the digitized signalcharacteristics encapsulated in the previous good packet; determiningthe error pattern of digitized signals received over a predeterminedtime frame; and selecting the error concealment algorithm based on inputcollected from the maximum interval, mean interval, last interval, firstinterval, maximum burst error or dropped digital signal, mean bursterror or dropped digital signal, last burst interval, and the firstburst interval.

A fourth error concealment method according to the present inventionenabling he receiver to determine the error concealment algorithm beforereceiving a data packet encapsulating a digital audio signal. Thismethod comprises steps of receiving packet loss rate feedback reportsfrom sets of active receivers of data packets encapsulating digitizedsignals; generating packet loss statistics based on the format forreporting and measuring the occurrence of error packets based thepattern description method; and transmitting the statistics to receiversthat have or are about to receive the digitized signal stream deliveredin a sequence of data packets.

A fifth error concealment method is a method enabling the selector todynamically select different error concealment methods for burst loss.This method comprises a step for switching the error concealment methodfrom a first method to a second method when the mean sampling amplitudeis less than a threshold T; a step for switching muting so that byreducing the amplitude of consecutive error-concealed error packets, theaudio or voice amplitude in playback packets is gradually reduced andfinally muted; and a merging step for applying packet merging betweenmultiple packets at the beginning and end of burst loss in order toprovide a smooth transition from a good packet to a concealed packet orfrom a concealed packet to a good packet.

This switching step preferably includes steps of: initializing a tablecomprising a list of threshold values; mapping each threshold value toan error concealment algorithm; computing the threshold T value based onerror loss pattern and environmental conditions; comparing said computedthreshold value T with the values in the initialized list of thresholdvalues; and selecting the algorithm that is closest to the computed Tvalue.

Further preferably, the muting switching step comprises steps of:setting silence threshold ST; multiplying each sample signal within thepacket with a constant C having a value less than 1; computing theaverage power P for said sampling signal; comparing average power P withsilence threshold ST; and switching to a mute mode when average power Pfalls below threshold ST.

Further preferably, the merging switching step comprises steps of:defining a template window W1 comprising n sample signals; multiplyingthe n samples of concealed signals at the burst boundaries with samplesin said template window W1; normalizing the product signals; andsubstituting the normalized signals for error-concealed packet segments.

A sixth method according to the present invention is a method formerging boundaries at the beginning and end of burst loss when the nsamples of error-concealed signals at the boundary are silence. Thismethod comprises steps of: defining a template window W2 comprising nsample signals; normalizing n signals in template window W2 to the nsignals at the boundary segment of a good packet; and substituting thenormalized samples for the segment of concealed packets at theboundaries.

A packet receiver according to the present invention comprises a packetreceiving means for receiving data packets from a network, detectingpacket loss in the received data packets, and outputting the detectionresult; the above-described selection means for selecting the best errorconcealment method for the dropped data packets and applying errorconcealment processing based on the detection result from the packetreceiving means; and a conversion means for converting theerror-concealed digital signal output from the selection means to ananalog signal.

Preferably, the packet receiving means of this packet receiver adds apacket header containing an error indicator field containing an errorindicator flag and a sequence number field containing a consecutivesequence number for each received packet to the received data packets,and then outputs the data packets.

Preferably, the selection means determines that packets having thispacket header and a payload of packet-size data following this packetheader are good packets if the error indicator flag denotes “no loss”and the sequence number field is in the correct sequence.

Further preferably, the selection means determines that packets havingthis packet header but not having a payload of packet data following thepacket header is a loss packet (dropped packet) if the error indicatorflag denotes a “loss packet” and the sequence number field is in thecorrect sequence.

A seventh error concealment method according to the present invention isa method for determining the consecutive burst loss count. This methodcomprises steps of: obtaining the packet headers and reading the errorindicator fields; recording the sequence number x when the burst startis encountered; recording the sequence number y when the burst end isencountered; and computing the number of consecutive packets in theburst loss as (y-x)+1 packets.

Preferably, this method for detecting the start of burst loss comprisessteps of: detecting that the current packet is a loss packet; anddetecting that the previous packet is a good packet.

Further preferably, the method of detecting the end of burst losscomprises steps of detecting that the current packet is a loss packet;and detecting that the following packet is a good packet.

A method according to an eighth embodiment of the invention selectsaudio error concealment algorithms without storing a long period ofaudio data for analysis using error statistics generated by the audiopacket receiver or other reliable network Anode capable, of collectingnetwork statistics and generating error pattern format based on thesecond method described above.

Other objects and attainments together with a fuller understanding ofthe invention will become apparent and appreciated by referring to thefollowing description and claims taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a dynamic error concealment selectoraccording to the present invention;

FIG. 2(a) shows the header and data fields in a valid packet, and FIG.2(b) shows the fields in a dropped packet (containing only a header andno data);

FIG. 3(a) to (d) show the signal at major nodes in the error concealmentselector show in FIG. 1;

FIG. 4(a) to (d) describe different algorithms;

FIG. 5 is a flow chart of an error concealment selection processaccording to the present invention;

FIG. 6 is a flow chart of a method for determining the burst loss count;

FIG. 7 is a flow chart of a selection process for processing the first nconsecutive packets;

FIG. 8 is a flow chart of a selection process for processing the next pconsecutive packets;

FIG. 9 shows part of a stream having eight consecutive dropped packets;

FIG. 10 is a flow chart of a mute switching process;

FIG. 11 is a flow chart of a merging process when the error-concealedpacket is not replaced with silence;

FIG. 12 is a flow chart of a merging process when the error-concealedpacket is replaced with silence;

FIG. 13 showing conversion of an error pattern detected in an audiopacket buffer to a numerical statistic used for error concealment; and

FIG. 14 is a block diagram of an audio packet receiver having an errorconcealment selector module according to the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention are described below withreference to the accompanying figures.

The error concealment apparatus shown in FIG. 1 has a packet lossdetector 1, packet audio buffer 2, loss pattern detector 3, amultiplexer 4 that functions as a digital audio payload manager, asignal analyzer 7, an error concealment selector 8, and an algorithmholder 9. The signal analyzer 7 has a voice detector 5 and an energycalculator 6. These components 1 through 9 are controlled by a computer10.

Operation of this error concealment apparatus is described next.

A voice signal (payload) as shown in FIG. 3(a) is carried in packet 1.This voice signal is a voice signal such as carried by telecommunicationnetworks. The voice signals are processed by pulse code modulation (PCM)and divided between multiple packets for transmission. In the exampleshown in FIG. 3(a), the voice signal is split and transmitted in 14packets. Note that packets 003 to 010 of these fourteen packets havebeen dropped in this example. Operation of this error concealmentapparatus is therefore described assuming that packets have been droppedfrom the voice signal input to the packet loss detector 1.

The packet loss detector 1 detects the packet sequence numbers, anddetermines if any packets have been dropped by detecting whether thesequence numbers are consecutive. The packet loss detector 1 adds aerror indicator flag set to 0 to the beginning of every valid packetreceived. If packet loss is detected, a substitution packet isgenerated, the dropped sequence number is added to the substitutionpacket, and the error indicator flag is set to 1 and added to thebeginning of the packet.

As shown in FIG. 2(a) and (b), the header of the packet output from thepacket loss detector 1 includes an error indicator field and a sequencenumber field. Valid packets and good packets contain a header andpayload. As shown in FIG. 2(a), the payload contains a PCM signal 23. Asubstitution packet (or loss packet) contains only the header and nopayload. The error indicator flag is set to 0 if the packet is errorfree, but is set to 1 in loss packets. The sequence number fieldcontains a sequence number that is consecutive through both validpackets and loss packets.

FIG. 3(b) shows the output signal from the packet loss detector 1, thatis, the input signal after packet loss detection and substitutionpackets have been added.

The packet audio buffer 2 takes the signal shown in FIG. 3(b) from thepacket loss detector 1, buffers and delays the signal, and thendistributes the packets based on the flag settings. This delay dependson the size of the PCM signals in the packets arriving at the receiver.Valid packets with the error indicator flag set to 0 are simply passedfrom the packet audio buffer 2 to the multiplexer 4 and to the algorithmholder 9.

These error indicator flags can be deleted in the output from the packetaudio buffer 2.

The packet audio buffer 2 outputs only the header (containing at leastthe error indicator flag and sequence number) from each packet to theloss pattern detector 3. The loss pattern detector 3 therefore receivesthe flags and corresponding sequence numbers. For the signal shown inFIG. 3(b), the loss pattern detector 3 receives a flag pattern signalwith the flag pattern 00111111110000, and the sequence number of reachflag (i.e., numbers 001 to 014 in this example).

The packet audio buffer 2 outputs only the PCM data payload of thepacket from which the headers have been stripped to the signal analyzer7. Note that the packets could be passed with the header included.

The signal analyzer 7 receives the flag pattern signal from the losspattern detector 3, and detects the PCM data in the valid packets with aerror indicator flag at the point in the flag pattern where the flagchanges from 0 to 1. In the example shown in FIG. 3(b), the signalanalyzer 7 therefore gets the PCM data from signal PCM2. The PCM signallatched by the signal analyzer 7 is then used as the template signal forsignal analysis. The template signal can be a short signal composed ofthe PCM signal from one

-   -   valid packet, or a longer signal composed of the PCM signals        from multiple valid packets.

The voice detector 5 detects whether the input template signal (signalPCM2) is a voice signal or non-voice signal. More specifically, thevoice detector 5 detects if the voice signal is a signal from atelephone conversation or is some other signal. A method such asdescribed in reference [6] cited above can be used for thisdetermination. The voice detector 5 therefore outputs a signal denotingwhether the template signal is a voice signal or non-voice signal.

The energy calculator 6 determines and outputs the PCM signal energylevel, that is, it determines the amplitude of the demodulated PCMsignal and outputs a signal value denoting the average amplitude W ofthe demodulated PCM signal.

The algorithm holder 9 stores different algorithms used to generate theinserted PCM signals used in the substitution packets. Four suchalgorithms are shown by way of example in FIG. 4.

Algorithm₁ shown in FIG. 4(a) is an algorithm for generating the PCMsignal of a dropped packet using the PCM signal of the valid packetimmediately preceding the dropped packet. More specifically, thisalgorithm defines a window W0 covering a last part or all of the voicesignal demodulated from the PCM signal of the valid packet, and expandsthis window to include both this window W0 and the voice signal part ofthe next dropped packet. This expanded voice signal is then modulated toa PCM signal again, producing the substitution packet. This algorithm iscalled “pitch waveform substitution.”

Algorithm₂ shown in Fig. 4(b) copies the PCM signal of the valid packetimmediately before the dropped packet to generate a substitution packet.This algorithm is called “packet repetition.”

Algorithm₃ shown in FIG. 4(c) substitutes a packet replacing the PCMsignal of the dropped packet with a PCM signal for white noise. Thisalgorithm is called “noise substitution. ”Algorithm₄ shown in FIG. 4(d)substitutes a packet replacing the PCM signal of the dropped packet witha PCM signal producing silence. This algorithm is called “silencesubstitution.”

Multiple algorithms are thus stored in the algorithm holder 9, and eachof the algorithms generates a PCM signal for the dropped packet andoutputs a substitution packet.

Which of these multiple algorithms is used is determined by theselection signal output by the error concealment selector 8. Operationof this error concealment selector 8 is described next.

The error concealment selector 8 receives the flag pattern signal fromthe loss pattern detector 3, the voice/non-voice signal and energyinformation from the signal analyzer 7, and environmental statusinformation indicating CPU usage from the computer 10. Parameter t1 inTable 3 indicates the usage of the computer's CPU. If CPU usage is 70%or more, parameter t1 is set to 0; if CPU usage is 50% to 70%, parametert1 is set to 0.15; if CPU usage is 30% to 50%, parameter t1 is set to0.3; and if CPU usage is less than 30%, parameter t1 is set to 0.5.[NOTE: TABLE 3 IS WRONG]

As shown in Table 4, parameter t2 denotes memory usage in the computer.As shown in Table 4, if memory usage is 70% or more, parameter t2 is setto 0; if memory usage is 50% to 70%, parameter t2 is set to 0.15; ifmemory usage is 30% to 50%, parameter t2 is set to 0.3; and if memoryusage is less than 30%, parameter t2 is set to 0.4. [NOTE: TABLE 4 ISWRONG]

These parameters t1 and t2 are then added and the sum is used asthreshold T.

The error concealment selector 8 determines from the signal suppliedfrom the, voice detector 5 whether the PCM signal of the substitutionpacket to be generated should be a voice signal or non-voice signal.More specifically, if the valid packet preceding the dropped packet is avoice signal, the PCM signal is generates to be voice signal, but if thepreceding packet was a non-voice signal, then the PCM signal isgenerated as a non-voice signal.

If a voice signal is to be generated, whether pitch waveformsubstitution or packet repetition is to be used is selected as shown inTable 1. More specifically, if the threshold T is 0.4 or greater, asignal selecting pitch waveform substitution, that is, algorithms, isoutput. However, if threshold T is less than 0.4, packet repetition,that is, algorithm₂ is selected.

If a non-voice signal is to be generated, whether noise substitution orsilence substitution is to be used is selected as shown in Table 2. Ifthe average energy W is 0.3 or greater, a selection signal for selectingnoise substitution, that is, algorithm₃, is output. If the averageenergy W is less than 0.3, then silence substitution, that is,algorithm₄, is selected.

The algorithm selection signal selected as described above is thenoutput to the algorithm holder 9, which selects the algorithmaccordingly. The PCM signal of the substitution packet is then generatedbased on the selected algorithm, a header containing the correspondingsequence number is added, and the result is output to multiplexer 4. Asa result, a substitution packet generated as shown in FIG. 3(d) isoutput from the algorithm holder 9, and the valid packets shown in FIG.3(c) are output from packet audio buffer 2, to the multiplexer 4.

The multiplexer 4 then arranges the errorless valid packets, and thegenerated substitution packets, in sequence number order, and outputsthe packet sequence to a digital/analog converter.

If the number of selectable algorithms is small, such as only two,selection could be based on only signals from, the voice detector 5, orbased only on signals from the energy calculator 6, or based only onthreshold T.

FIG. 5 is a flow chart of the process for selecting the errorconcealment method.

The first step is acquiring the packet header from the packet audiobuffer 2 (step S1). Whether another packet is in the packet audio buffer2 is then checked (step S2). If another packet is not in the packetaudio buffer 2, the selection process ends. Otherwise, the flag in theerror indicator field is read (step S3). If the error indicator flag is1, the current packet is recognized as a substitution packet (v2 (b)).If the error indicator flag is 0, the packet is recognized as a validpacket (FIG. 2 (a)). (Case 1: a substitution packet is received)

Error concealment is then applied. First information about currentresource usage is checked by calculating current CPU usage and memoryusage (step S5). The number of consecutive substitution packets is thencounted (step S6) by the loss pattern detector 3. A method of countingthe substitution packets is described next with reference to the flowchart in FIG. 6.

The first substitution packet is first detected (step S21). This is doneby reading the error indicator field in each packet to find the firstsubstitution packet having the error indicator flag set to 0 after avalid packet in which the error indicator flag is set to 1. When thebeginning of a substitution packet sequence is found, the sequencenumber x of that first substitution packet is stored (step S22). The endof the substitution packet sequence is then detected (step S23). This isdone by again reading the error indicator flags to find the currentpacket that is a substitution packet followed by a valid packet. Whenthe end of the substitution packet sequence is found, the sequencenumber y of that packet is saved (step S24). The number of consecutivesubstitution packets is then obtained as (y-x)+1 (step S25).

Referring back to FIG. 5, the substitution packet X sequence detected instep S6 is processed. The substitution packets are processedindividually. More specifically, the error concealment techniques candiffer from packet to packet.

Whether all substitution packets have been processed is then determined(step S7), and whether all n packets have been processed is determined(step S8). The first n substitution packets in the consecutivesubstitution packet sequence are then processed (step S9) where n iscalculated from the number of consecutive substitution packets.

Details of processing the first n packets is described with reference tothe flow chart in FIG. 7.

After these n packets are processed, the remaining packets (p packets inthis example) are processed (step S10). Details of processing these ppackets are described with reference to the flow chart in FIG. 8.

Case 2: Valid Packets Received

When a valid packet is received, the data segment 23 shown in FIG. 2(a)is buffered and sent to the multiplexer 4 (step S4). If the packetfollowing this valid packet is determined to be a substitution packet,the valid packet is used as a template signal by the signal analyzer 7.

Processing the first n substitution packets is described with referenceto FIG. 7. First, the valid packets before and after the n substitutionpackets are sent to the voice detector 5 and stored for use as templatesignals (step S31). The voice detector 5 ‘then ’ detects whether the PCMsignal of that valid packet is a voice signal or non-voice signal (stepS32). If the valid packet is a voice signal, the threshold T iscalculated from the current operating conditions, including CPU usageand memory usage, as described above. This threshold T is then comparedwith the threshold value T1 in the list (Table 1) of specific errorconcealment techniques for processing voice signal (step S35). The errorconcealment method is determined by finding the range in Table 1 inwhich this threshold T belongs (steps 36), and error concealment is thenapplied based on the selected algorithm.

If the valid packet is a non-voice signal in step S32, the averageenergy W is calculated for all signals (step S33). The error concealmentmethod is selected from the non-voice algorithm list (Table 2) based onthe average energy W (step S34). More specifically, the average energy Wis compared with threshold value T2. If the average energy W is lessthan T2 (that is, a value of 0 to 0.3 in Table 2), it can be ignored andsilence substitution is used. However, if the average energy W isgreater than threshold value T2 (exceeds 0.3 in Table 2), noise havingaverage energy similar to white noise is generated and used as thesubstitution packet.

If the sequence of substitution packets is long, signal amplitude isgradually reduced in the plural p packets substituted after insertingthe first n packets until a mute effect is achieved.

The dynamic selector described here can select from four errorconcealment methods. These four error concealment methods include twofrom the voice list and two from a non-voice list.

The voice list includes the following methods. TABLE 1 Algorithmselection for voice signals Error Concealment method Threshold, T1 (0-1)Pitch Waveform substitution ≧0.4 (algorithm 1) Packet repetition(algorithm 2) 0-0.4

The non-voice list includes the following methods. TABLE 2 Algorithmselection for non-voice signals Error Concealment method Threshold, T2(0-1) Noise substituion (algorithm 3) ≧0.3 Silence substitution(algorithm 4) 0-0.3

When a consecutive sequence of substitution packets is received asdetected by step, S6 in FIG. 5, such as when eight consecutivesubstitution packets are received as shown in FIG. 9, n is set asdetermined by equation 1 below.0.3*missing_packet_count=3  (1)

The first n packets, that is, three packets in this example, areprocessed by the method shown in FIG 7. The first valid packet beforethese three substitution packets is stored in the signal analyzer 7 asthe template signal. This template signal is first sent to the voicedetector 5 to determine whether it is a voice signal or non-voice signal(step S31). In this example the template signal is a voice signal, andthe voice detector 5 returns the voice flag (step S32). Current resourceusage is then checked. The threshold T is then calculated based on twostandard values, specifically CPU usage CPC and memory usage MEC. If CPUusage CPC exceeds 70%, threshold parameter t1 is set to 0. If memoryusage MEC exceeds 70% of all available memory, threshold parameter t2 isset to 0. These threshold parameters are calculated as shown below.TABLE 3 Setting the CPU usage parameter t1 CPU usage (CPC) Parameter t1CPC >= 70% 0 70% > CPC >= 50% 0.15 50% > CPC >= 30% 0.3 30% > CPC 0.5

TABLE 4 Setting memory usage parameter t2 Memory usage: MEC Parameter,t2 MEC >= 70% 0 70% > MEC >= 50% 0.15 50% > MEC >= 30% 0.3 30% > MEC 0.4

Threshold T is calculated from the following equation (2).T=t1+t2  (2)

If current CPU usage is 40% and memory usage is 50%, threshold T is0.45. Because the previous valid packet is a voice signal, threshold Tis compared with T1 (step S35). If the previous valid packet is anon-voice signal, average energy W is compared with T2. In this casethreshold T could be compared with T2 instead of comparing averageenergy W. The selection criteria is that T exceeds either T1 or T2, andthe error concealment method with the closest value is selected. Becausethreshold T is 0.45 in this case, pitch waveform substitution isselected (step S36). The waveform of the first three substitutionpackets is therefore concealed using pitch waveform substitution.

Subsequent substitution is done using the algorithm shown in FIG. 8 anddescribed below.

The value of (X−n) shown in step S41 in FIG. 8 is, in this example,(8-3)=5. More specifically, X is the number of consecutive substitutionpackets, n is the first packet, and the difference is the number ofsubstitution packets on which the same process is repeated.

Table 5 shows the relationship between the previously selected errorconcealment method and the error concealment method selected next. Ifthe error concealment method applied to the previous substitution packetis pitch waveform substitution, the error concealment method applied tothe next substitution packet is packet repetition with amplitudemultiplied C times (C<1). If the error concealment method of theprevious substitution packet is packet repetition, the error concealmentmethod of the next substitution packet is also packet repetition withamplitude multiplied C times (C<1). If the error concealment method ofthe previous substitution packet is noise substitution, the errorconcealment method of the next substitution packet is also noisesubstitution. If the error concealment method of the previoussubstitution packet is silence substitution, the error concealmentmethod of the next substitution packet is also silence substitution.

In the above example pitch waveform substitution is applied to theprevious substitution packet. Packet repetition is therefore applied tothe next packet, as well as to the next packet after that. Packetrepetition thus continues with a gradual reduction in amplitude. Errorconcealment thus repeats five times (step S43). Packet substitution isthus applied packet by packet: until all remaining five packets areconcealed (step S44). Packet processing ends after the fifth packet isconcealed. TABLE 5 Next packet substitution Previous Next Pitch WaveformPacket Repetition (amplitude * C) Packet Repetition Packet Repetition(amplitude * C) Noise Substitution Noise Substitution SilenceSubstitution Silence Substitution (muting) (muting)

When substitution repeats as described above, substitution energygradually reduces, and when the average energy becomes less than thesilence threshold ST, silence substitution is used for all subsequentsubstitutions.

FIG. 10 is a flow chart of the muting switching process. This mutingswitching process includes, for example, a step setting the silencethreshold ST, multiplying each of the sampling signals in a packet by aconstant C with a value less than 1 (step S51), multiplying the samplingsignal from step S51 by the average energy P of the packet or part ofthe packet (step S52), comparing average energy P and silence thresholdST (step S53), and switching to a muting mode when the average energy Pis less than silence threshold ST (step S54).

Merging can be used to assure a smooth transition at the boundarybetween a valid packet and substitution packet. If the difference inaverage energy at this boundary is less than a specific value Y, mergingis not needed. Otherwise, merging is necessary. Merging is done by themultiplexer 4.

A merging process is described next with reference to FIG. 11 and FIG.12. FIG. 11 is a flow chart of the merging process when silencesubstitution was not used for packet error concealment.

A predefined template window W is composed of n signal samples. The lastpacket in a consecutive valid packet sequence is handled in window We.The first packet in a consecutive substitution packet sequence followingthe valid packet sequence is handled in window Ws: (step S61).

Windows We and Ws therefore indicate the boundary between a valid packetand a substitution packet. The average energy in window We is thenobtained. The energy in window Ws is normalized by this average energy(step S62). The sample signals in window Ws are replaced so that thereis a smooth energy change between windows We and Ws (step S63).

FIG. 12 is a flow chart of the merging process used when the errorconcealment packets are replaced with silence. Template window W2 isdefined only for using silence substitution. Template window W2 isnormalized to the value of the mean energy of the boundary segmentsbetween good packets (step S71). The normalized samples then directlyreplace the boundary segments of the error concealment packets (stepS72).

The algorithm selection process is based on information generated fromnetwork performance statistics.

These statistics are distributed over the network in a format similar tothe Pattern_Loss structure by a network node capable of generating suchinformation. This information could also be generated based on thepacket audio loss pattern received by the packet audio buffer over anextended time. This selection process is based on long term analysis,which could also be defined as the time interval of an error losspattern exceeding the time interval of audio buffered to the packetaudio buffer. For error concealment over an extended time, the dataelements in the Pattern_Loss structure can be used for error concealmentapplied to audio data received by the packet audio buffer 2 in FIG. 1.

More specifically, short term analysis in which only consecutive packetsentering the packet audio buffer 2 are examined to select the errorconcealment method is described above. Long term analysis in which theerror concealment method is selected using the results of continuouslychecking packets entering the packet audio buffer 2 for a specified longtime (the buffer frame period shown in FIG. 13) is described below. Inlong term analysis, the data values (1) to (9) described below areupdated each time the buffer frame period passes.

The loss pattern detector 3 executes either short term analysis or longterm analysis.

So that error concealment can be done in the same way in an audiobroadcast environment, these statistics in the form of the Pattern_Lossstructure enable error concealment to be performed in a unified manner.Different receivers do not necessarily start receiving the samebroadcast at the same time, but the different receivers can select andapply the algorithm. If the receiver has little available computingcapacity, the loss pattern detector 3 can be turned off. Selecting theerror concealment method by means of long term analysis depends on thelimited loss pattern observed by the packet audio buffer 2 (that is, theflag pattern described above), and the statistical mean (data values (1)to (9) described below) of the loss pattern generated by the losspattern detector 3 in FIG. 1.

It should be noted that the algorithm could be selected using theresults of both long term analysis and short term analysis, or onlyusing the results of either long term analysis or short term analysis.

The error statistics generated by the loss pattern detector 3 are in theformat of the structure shown below measured over a buffer frame period.The buffer frame period used to determine the temporal loss pattern andthe period is defined as a unit time, or as a block of a digital audiosignal composed of many digital audio data packets. It is more effectivefor the buffer frame period to be longer than the packet period storedin the packet audio buffer. Pattern_Loss { (1) Delay; /* Units inseconds or packet payload size       (bytes/bits) */ (delay 52 of thesize of the       PCM signal in one packet on the time base) (2) MeanLoss Interval; /* average time interval between 2       dropped packets*/ (average of L_I) (3) Max Loss Interval; /* maximum time intervalbetween 2       dropped packet */ (L_I(max) 54) (4) First Loss Interval;/* time interval measured from the       beginning of the buffer frameto the beginning       of Packet loss */ (L_I(p) 53) (5) Last LossInterval; /* time interval measured from the       first packet of thelast group of non-erroneous       packets to the end of the buffer frame*/       (TTL1) (6) Mean Burst Interval; /* average time intervalbetween 2       non-erroneous or loss packets */ (average of       B_L)(7) Max Burst Interval; /* maximum time interval between 2      non-erroneous or loss packets */ (B_L(max)) (8) First BurstInterval; /* first group of consecutive       erroneous or loss packetsmeasured from the       beginning of the buffer frame to the end of the      last packet of the group of error packets */       (FBL) (9) LastBurst Interval; /* Time interval measured from the       first packet ofthe last group of error packets       to the end of the buffer frame */(LBI) }

FIG. 13 illustrates the method used in determining, and describing theerror pattern by generating statistics from the Pattern_Loss structureas shown above.

Buffer frame 51 has an arbitrary length determined by the number ofaudio data packets in which the error pattern is to be determined. Thepacket length in this example is fixed and is determined by Delayparameter 52. The Mean Loss interval is the average of all lossintervals measured within the Buffer Frame. In this example:Mean Loss Interval={L _(—) I(max)+L _(—) I+L _(—) I(p)}/total intervalof error-free packets.

The mean burst interval is the average of all burst intervals measuredwithin the Buffer Frame. In this example:Mean Burst Interval={B _(—) L(max)+B _(—) L +B _(—) L(n)}/total intervalof error packets;

Variable L_I(p) 53 gives the value of the First Loss Interval element ofthe Pattern_Loss structure.

Variable L_I(max) 54 gives the value of the Max Loss Interval element ofthe Pattern_Loss structure. Variable L_155 is one data value used tomeasure the loss interval within the Buffer Frame.

The Last Burst Interval element of the Pattern_Loss structure isequivalent to variable B_L(n) 56.

Variable B_L 57 is one of the burst error values measured within thebuffer frame.

Variable B_L(max) 58 is the equivalent of the Max Burst Interval elementof the Pattern_Loss structure for the Buffer frame shown by way ofexample in FIG. 13.

The following Table 6 shows a number of examples of how the error lossstatistics derived from the Packet_Loss structure can be used by theerror concealment selector to make a better decision on selecting theerror concealment algorithm to produce an error-free audio stream withoptimal error concealment. The error concealment is applied to the audiopackets stored in the packet audio buffer 2 shown in FIG. 1. In somecases, the packet audio buffer 2 can only store audio signals for ashort period. The specific time interval of the audio signals and theloss time interval presented here do not have specific numeric valuesbut instead uses a high and low representation for relative comparison.For detail implementation, it is necessary to know the packet size andthe sampling rate of the audio signals and the type of error concealmentalgorithm available. TABLE 6 Use of error loss statistics by the errorconcealment selector Example of Error Concealment s/n Packet_Loss ValueAlgo Used Description 1 LBI > specified Muting Due to excessive loss,value muting is best method. 2 (Mean Burst Lost Pitch waveform Othererror concealment Interval < specified substitution or methods thatperform better value) Packet replication in a low bursty error AND (LBI= 0) environment. 3 [B_L(max) < Time scale Other Error concealmentspecified value] modification methods capable of AND extending the timescale [(L_I) (max) > have close to undetectable specified value] erroroutput. AND LBI = 0

Note that the Pattern_Loss structure described above is just one exampleof statistical analysis that can be used to describe the error losspattern, and the invention shall not be limited thereto. Furthermore,the buffer frame period and packet loss interval can also be describedby bit count rather than time.

FIG. 14 is a block diagram of an audio packet receiver comprising an,error concealment selector module according to the present invention.The receiver 70 shown in FIG. 14 has an audio packet receiver unit 71,audio packet decoding unit 72, error concealment selection module 73,and audio output unit 74.

The audio packet receiver unit 71 receives audio packets from a networkand detects whether any audio packets were dropped in the receivedstream. If audio packet loss is detected, the detection information ispassed to the audio packet decoding unit 72.

If the audio packets output from the audio packet receiver unit 71 areencoded, the audio packet decoding unit 72 decodes the packets toproduce a PCM audio stream.

The error concealment selection module 73 generates the needed PCMpackets. Note that this error concealment selection module 73 is asshown in the block diagram in Fig.

The audio packet decoding unit 72 also receives the packet lossinformation detected by the audio packet receiver unit 71, and outputsthis packet loss information to the error concealment selection module73.

The error concealment selection module 73 determines whether the packetsreceived from the audio packet decoding unit 72 are good packets or losspackets, and executes packet processing accordingly as described above.

If a packet contains the packet header added by the audio packetreceiver unit 71 and a payload following the header, the error indicatorflag added to the header denotes a good packet (i.e., not a droppedpacket), and the packet sequence number in the sequence number field isin the correct order, the error concealment selection module 73recognizes the packet as good packet. If the error indicator flag in thepacket header added by the audio packet receiver unit 71 denotes adropped packet, and the packet sequence number in the sequence numberfield is in the correct order, the error concealment selection module 73recognizes that packet as an error packet.

The audio output unit 74 converts digital audio processed and output bythe error concealment selection module 73 to analog audio, and outputsthe result from a speaker.

The audio packet decoding unit 72 can be omitted when the audio packetsoutput from the audio packet receiver unit 71 is unencoded PCM audio.

This invention allows a combination of different error concealmentmethods to be used at different times. The most appropriate choice canbe selected dynamically at run-time. An error concealment algorithm withless complexity can therefore be used when resources are low, and a moreeffective but more complex algorithm can be chosen if sufficientresources are available.

The algorithms are selected from a set of pre-defined algorithms withdifferent characteristics. In the case of voice classes. The errorconcealment algorithm applied to voice segments can then be chosen fromone list of methods, and the error concealment algorithm applied tonon-voice segments can be chosen from another list of methods. For burstloss in which many consecutive packets are missing, this algorithmselector enables, the burst loss to be concealed using different methodswith different packets in the burst loss.

The invention also enables an audio receiver with a small audio buffersize and low latency to implement complex error concealment algorithmsby using error statistics to measure the error pattern in packetizedaudio streams. Based on the error statistics collected, the errorconcealment selector can make better decisions regarding the type oferror concealment to be used for particular audio loss frames, andthereby output an audio stream with better error concealment.

Although the present invention has been described in connection with thepreferred embodiments thereof with reference to the accompanyingdrawings, it is to be noted that various changes and modifications willbe apparent to those skilled in the art. Such changes and modificationsare to be understood as included within the scope of the presentinvention as defined by the appended claims, unless they departtherefrom.

1. An error concealment apparatus for detecting dropped packets in anaudio signal composed of multiple packets, and generating substitutionpackets for concealing the dropped packets, the apparatus comprising: apacket loss detector for detecting dropped packets in a received streamof valid packets; an extractor comprising a buffer and extracting avalid packet before a dropped packet; an analyzer for analyzing audiosignals in the extracted valid packets; an algorithm memory for storinga plurality of algorithms and generating a substitution packet using oneof the algorithms; a selector for determining which of the plurality ofalgorithms to select using the result from the analyzer; and amultiplexer for inserting the substitution packet at the location of thedropped packet in the valid packet stream.
 2. An error concealmentapparatus as described in claim 1, further comprising: a CPU foroperational control; memory operating in conjunction with the CPU; and aresource availability calculator for measuring available CPU and memorycapacity; wherein the selector uses the calculated available capacity todetermine which of the algorithms to use.
 3. An error concealmentapparatus as described in claim 1, wherein the analyzer comprises avoice detector for determining if the audio signal contains a voicesignal or silence.
 4. An error concealment apparatus as described inclaim 1, wherein the analyzer comprises an energy calculator forcalculating audio signal energy.
 5. An error concealment apparatus asdescribed in claim 1, wherein the packet loss detector adds to allpackets that should be received, a flag for identifying valid packetsand dropped packets.
 6. An error concealment apparatus as described inclaim 5, wherein the extractor extracts the flags and generates a flagpattern in a flag sequence.
 7. An error concealment apparatus asdescribed in claim 6, wherein the selector extracts the flag pattern,and based on flag pattern analysis determines which of the multiplealgorithms to select.
 8. An error concealment apparatus as described inclaim 1, wherein the plurality of algorithms include: a first algorithmfor correcting the audio signal in the valid packet preceding a droppedpacket to create an audio signal for the dropped packet; a secondalgorithm for copying the audio signal of the valid packet preceding adropped packet to create the audio signal of the dropped packet; a thirdalgorithm for creating the audio signal of a dropped packet using whitenoise; and a fourth algorithm for creating the audio signal of a droppedpacket using a silence signal.
 9. An error concealment method fordetecting dropped packets in an audio signal composed of multiplepackets, and generating substitution packets for concealing the droppedpackets, the method comprising: detecting dropped packets in a receivedstream of valid packets; extracting a valid packet before a droppedpacket; analyzing audio signals in the extracted valid packets; storinga plurality of algorithms and generating a substitution packet using oneof the algorithms; determining which of the plurality of algorithms toselect using the result of the analyzing; and inserting the substitutionpacket at the location of the dropped packet in the valid packet stream.10. An error concealment method as described in claim 9, furthercomprising: for measuring available capacity in a CPU (10) foroperational control and memory (10) operating in conjunction with theCPU; and using the calculated available capacity to determine which ofthe algorithms to use.
 11. An error concealment method as described inclaim 9, wherein the analyzing comprises determining if the audio signalcontains a voice signal or silence.
 12. An error concealment method asdescribed in claim 9, wherein the analyzing comprises calculating audiosignal energy.
 13. An error concealment method as described in claim 9,further comprising adding to all packets that should be received, a flagfor identifying valid packets and dropped packets.
 14. An errorconcealment method as described in claim 13, further comprisingextracting the flags and generating a flag pattern in a flag sequence.15. An error concealment method as described in claim 14, furthercomprising a step for extracting the flag pattern, and based on flagpattern analysis, determining which of the multiple algorithms toselect.
 16. An error concealment method as described in claim 9, whereinthe plurality of algorithms include: a first algorithm for correctingthe audio signal in the valid packet preceding a dropped packet tocreate an audio signal for the dropped packet; a second algorithm forcopying the audio signal of the valid packet preceding a dropped packetto create the audio signal of the dropped packet; a third algorithm forcreating the audio signal of a dropped packet using white noise; and afourth algorithm for creating the audio signal of a dropped packet usinga silence signal.